Discriminator-Enhanced Knowledge-Distillation Networks

Query auto-completion (QAC) serves as a critical functionality in contemporary textual search systems by generating real-time query completion suggestions based on user’s input prefix. Despite the prevalent use of language models (LMs) QAC candidate generation, LM-based approaches frequently suffer from overcorrection issues during pair-wise loss training and efficiency deficiencies. To address these challenges, this paper presents novel framework—discriminator-enhanced knowledge distillation (Dis-KD)—for task. This framework combines three core components: large-scale pre-trained teacher model, lightweight student discriminator for adversarial learning. Specifically, aids discerning generative-level differences between models. An additional score is amalgamated with traditional knowledge-distillation loss, resulting enhanced performance model. Contrary to stepwise evaluation each generated word, our approach assesses entire generation sequence. method alleviates issue process. Consequently, proposed boasts improvements model accuracy reduction parameter size. Empirical results highlight superiority Dis-KD over established baseline methods, surpassing tasks sub-word languages.